Enhancing tangible content on physical activity surface

ABSTRACT

A method for enhancing tangible content on a physical activity surface is described. In an example implementation, the method includes capturing, using a video capture device of a computing device, a video stream that includes an activity scene of a physical activity surface; detecting in the video stream, using a detector executable on the computing device, a tangible content item on the physical activity surface; recognizing, from the video stream, one or more visually instructive elements in the tangible content item; determining a tangible identifier based on the one or more visually instructive elements in the tangible content item; retrieving a digital content item using the tangible identifier; and providing the digital content item on the computing device.

BACKGROUND

The present disclosure relates to a computing device.

Computing devices are often used to access and view digital information.This digital information can be present within an application, viewed ina graphical user interface on an internet webpage, or even created bythe user, such as a document or image. This digital information isaccessible to the user via the computing device to allow the user tolearn from the digital information.

Physical information has historically been used to gain access toinformation. Physical information can be included in books, such astextbooks, encyclopedias, and other publications. A user may access thisphysical information by physical interacting with the book and viewingthe information on individual pages of the book.

If a user wants to access both physical information and digitalinformation that is related to each other, the user has to manuallyaccess the physical information, such as by opening and viewing thecontent of a book and also has to manually retrieve the digitalinformation, such as by accessing a webpage to view the webpagescontents. This results in an inconvenient and time intensive process tosupplement physical information with digital information. Existingsolutions create digital applications that present digital informationto the user and may link or reference to physical content, such as aspecific student workbook designed to be used with the digitalapplications, but these existing solutions are not able to intuitivelycombine the digital information experience and the physical informationexperience together beyond this limited use of specially designedsoftware applications and specific student workbooks or other speciallydesigned specific physical content.

SUMMARY

According to one innovative aspect of the subject matter in thisdisclosure, a method for enhancing tangible content on a physicalactivity surface is described. In an example implementation, a methodincludes capturing, using a video capture device of a computing device,a video stream that includes an activity scene of a physical activitysurface; detecting in the video stream, using a detector executable onthe computing device, a tangible content item on the physical activitysurface; recognizing, from the video stream, one or more visuallyinstructive elements in the tangible content item; determining atangible identifier based on the one or more recognized visuallyinstructive elements in the tangible content item; automaticallyretrieving a digital content item using the tangible identifier; andproviding the digital content item for display in a user interface ofthe computing device.

Implementations may include one or more of the following features. Themethod may include: detecting, in the video stream, a distinct visualelement on the tangible content item; and where determining the tangibleidentifier includes determining the tangible identifier based on thedistinct visual element on the tangible content item. The distinctvisual element includes one or more of an image, a drawing, a diagram,and a vision marker visible on the tangible content item. Determiningthe tangible identifier includes determining the tangible identifierbased on one or more of the document title, the page number, the sectiontitle, and the section number associated with the tangible content item.Determining the tangible identifier includes determining the tangibleidentifier based on the one or more characters associated with the usermarking. The user marking includes one or more of a highlighting effectand a user note created by a user on the tangible content item. Thetangible content item is a first tangible content item: the firsttangible content item is included in a tangible object that alsoincludes a second tangible content item; and the method includes:determining that a user interacted with the second tangible content itemof the tangible object; recognizing one or more visually instructiveelements in the second tangible content item; and determining thetangible identifier based on the one or more visually instructiveelements in the first tangible content item and the one or more visuallyinstructive elements in the second tangible content item of the tangibleobject. Determining the tangible identifier includes determining a firsttangible identifier and a second tangible identifier based on the one ormore visually instructive elements in the tangible content item; andretrieving the digital content item using the tangible identifierincludes: determining a contextual relationship between the firsttangible identifier and the second tangible identifier; and retrievingthe digital content item based on the contextual relationship betweenthe first tangible identifier and the second tangible identifier. Thecomputing device is placed on a stand situated on the physical activitysurface; and a field of view of the video capture device of thecomputing device is redirected by a camera adapter situated over thevideo capture device of the computing device. The tangible content itemon the physical activity surface includes a first tangible contentsection and a second tangible content section, the method may include:detecting, using a detector executable on the computing device, anarrangement of the first tangible content section relative to the secondtangible content section on the physical activity surface; determining asupplemental digital content item based on the arrangement of the firsttangible content section relative to the second tangible contentsection; and displaying in the user interface on the computing device,the supplemental digital content along with the digital content item.The first tangible content section is a first programming tile and thesecond tangible content section is a second programming tile, the firstprogramming tile being coupled with the second programming tile to formthe tangible content item. The supplemental digital content is aninstruction set for how to use the first programming tile and the secondprogramming tile. Displaying the supplemental digital content furthermay include: automatically displaying a first digital representation ofthe first programming tile in the user interface; automaticallydisplaying a second digital representation of the second programmingtile in the user interface; and automatically displaying an animation ofthe first digital representation interacting with the second digitalrepresentation to provide instructions on how to arrange the firstdigital programming tile and the second digital programming tile on theactivity surface. Providing the digital content item for display in theuser interface on the computing device further may include: generating avisualization of the digital content item for display in the userinterface; and displaying an animation of the digital content itemappearing in the user interface. The animation of the digital contentitem depicts a first portion of the visualization of the digital contentitem extending out from a bottom side of the display screen and a secondportion of the visualization of the digital content item appearing asthe animation causes the visualization of the digital content item tomove into the user interface. The animation of the digital content itemcorresponds to an appearance of the tangible content item being detectedon the physical activity surface. The visualization of the digitalcontent item is selectable by a user and responsive to the digitalcontent item being selected, the digital content item may be displayed.

The method also includes capturing, using a video capture device of acomputing device, a video stream that includes an activity scene of aphysical activity surface; detecting in the video stream, using adetector executable on the computing device, a tangible content item onthe physical activity surface, the tangible content item including apage visible within a field of view of the video capture device;determining, using a processor of the computing device, an identity ofthe page of the tangible content item from the video stream;determining, using the processor of the computing device, a concept ofthe tangible content item using the identity of the page; determining,using the processor of the computing device, a digital content itembased on the concept of the tangible content item and the identity ofthe page of the tangible content item; and presenting, in a display ofthe computing device, the digital content item in a graphical userinterface.

Implementations may include one or more of the following features. Themethod where determining the identity of the page of the tangiblecontent item further may include: detecting a page identity anchor onthe page; and matching the page identity anchor to a database of pageidentities. The tangible content item is a book that includes aplurality of pages and each of the plurality of pages has a unique pageidentity anchor. Determining the digital content item based on theconcept of the tangible content item and the identity of the page of thetangible content item further may include: identifying visuallyinstructive elements that are present on the page of the tangiblecontent; determining a subject context based on the visually instructiveelements; and identifying the digital content item based on the subjectcontext. Identifying the digital content item based on the subjectcontext further may include: determining a category level of a userbased on the concept of the tangible content item and the identity ofthe page of the tangible content item; accessing a node of a knowledgegraph that is related to the subject context, the node of the knowledgegraph including links to a plurality of digital content items related tothe subject context; and determining a relevant digital content itemfrom the plurality of digital content items based on the category levelof the user. Detecting a page identity anchor on the page further mayinclude: performing a matching search to identify the page identityanchor, the matching search including ignoring all text that does notinclude the page identity anchor. The digital content item is a videorelated to the identified page of the tangible content item. Presentingthe digital content item further may include: embedding an external linkto the video related to a topic present on the identified page of thetangible content item; and causing the external link to be executedresponsive to an input received from a user. The digital content item isone or more of a video, a question, a quiz, and a worksheet. Determiningan identity of the page of the tangible content item from the videostream further may include: detecting, using a processor of a computingdevice, a finger of a user present in the video stream of the physicalactivity surface; determining an area of the tangible content item thatis being pointed to by a point of the finger; determining a tangibleidentifier included within the area of the tangible content item; anddetermining an identity of the page of the tangible content item basedon the tangible identifier.

The visualization system also includes a computing device positioned ona physical activity surface; a video capture device coupled to thecomputing device, the video capture device including a field of viewdirected towards the physical activity surface and capable of capturinga video stream of the physical activity surface; a detector of thecomputing device configured to detect a tangible content item within thevideo stream and recognize one or more visually instructive elements inthe tangible content item; a processor of the computing deviceconfigured to determine a tangible identifier based on the recognizedvisually instructive elements in the tangible content item andautomatically retrieve a digital content item using the tangibleidentifier; and a display screen coupled to the computing device, thedisplay screen being configured to display the digital content item in agraphical user interface.

Other implementations of one or more of these aspects and other aspectsdescribed in this document include corresponding systems, apparatus, andcomputer programs, configured to perform the actions of the methods,encoded on computer storage devices. The above and other implementationsare advantageous in a number of respects as articulated through thisdocument. Moreover, it should be understood that the language used inthe present disclosure has been principally selected for readability andinstructional purposes, and not to limit the scope of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is illustrated by way of example, and not by way oflimitation in the figures of the accompanying drawings in which likereference numerals are used to refer to similar elements.

FIG. 1A illustrates an example tangible content item and an examplecomputing device that captures the tangible content item.

FIG. 1B illustrates an example tangible content item and an examplecomputing device that provides digital content items related to thetangible content item.

FIG. 1C is an example configuration for enhancing tangible content on aphysical activity surface.

FIG. 1D is an example configuration for enhancing tangible content on aphysical activity surface.

FIG. 2 is a block diagram illustrating an example computer system forenhancing tangible content on a physical activity surface.

FIG. 3 is a block diagram illustrating an example computing device.

FIG. 4 is a flowchart of an example method for enhancing tangiblecontent on a physical activity surface.

FIG. 5 is an example configuration for enhancing tangible content on aphysical activity surface.

FIG. 6 is a flowchart of an example method for enhancing tangiblecontent on a physical activity surface.

FIGS. 7A-7D are a graphical representation of an example user interfacefor enhancing tangible content.

FIG. 8 is an example configuration for finger detection to enhancetangible content.

DETAILED DESCRIPTION

The technology described herein is capable of providing data enhancementfor a tangible content item that is present on a physical activitysurface. As described in detail below, a computing device may capture avideo stream of the physical activity surface, and detect the tangiblecontent item in the video stream. The computing device may apply variousrecognition techniques on the tangible content item to recognizevisually instructive elements (e.g., characters, letters, numbers,symbols, etc.), distinct visual elements (e.g., images, drawings,diagrams, etc.), user markings (e.g., notes, highlights), etc., in thetangible content item. The computing device may determine one or moretangible identifiers based on the visually instructive elements, thedistinct visual elements, the user markings, etc., that are recognizedin the tangible content item, and retrieve digital content items usingthe one or more visually instructive elements, the distinct visualelements, the user markings, etc., and provide the digital content itemsto a user via the computing device. Thus, the present technology canprovide on the computing device the digital content items that arerelated to the tangible content item physically present on the physicalactivity surface, thereby effectively enhancing the content dataprovided to a user and improving user experience. In someimplementations, this process may happen automatically when a tangiblecontent item is detected and without any prompting or request by a user.

The technology described herein is advantageous, because it canfacilitate the user in obtaining additional data relevant to the contentof interest, especially in the context of education. As an example, thetangible content item may be a text book that is placed on the physicalactivity surface and opened to a page presenting content aboutairplanes. In this example, the computing device may provide digitalcontent items (e.g., images, video, online articles, research papers,etc.) related to the airplanes on the computing device. Thus, the usermay be provided with additional information about various aspects of theairplanes that is potentially useful or interesting but the user may notbe aware or may not think of. Furthermore, this presentation of thedigital content related to airplanes may happen automatically and allowfor the user to supplement their learning without having to do anythingother than open the page of the book.

FIGS. 1A-1D illustrate example configurations for enhancing a tangiblecontent item on a physical activity surface. As depicted in FIG. 1A, theconfiguration may include a tangible content item 102 situated on aphysical activity surface 150 and a computing device 100 including avideo capture device 110. The video capture device 110 may be adapted tocapture an image or a video stream that includes an activity scene 132of the physical activity surface 150.

In some embodiments, the tangible content item 102 may include one ormore portions that are exposed and visible to the video capture device110 of the computing device 100, and thus the video capture device 110can capture the visible portions of the tangible content item 102 in animage or in a video frame of a video stream. As an example, the tangiblecontent item 102 may be a book opened at one or more pages, and theportions of the tangible content item 102 may include the one or morepages of the book that are visually exposed to the video capture device110 of the computing device 100. In another example, the tangiblecontent item 102 may be a collection of paper sheets, and the visibleportions of the tangible content item 102 may be one or more papersheets in the collection that are visually exposed to the video capturedevice 110 of the computing device 100. In another example, the tangiblecontent item 102 may be an electronic device (e.g., tablet, mobilephone, etc.) displaying images and/or text in a graphical user interfaceof a display screen resting on the physical activity surface 150, andthe visible portions of the tangible content item 102 may be thedisplayed content items that are visually exposed to the video capturedevice 110 of the computing device 100. In another example, the tangiblecontent item 102 may be a content item that is projected by a projectoronto the physical activity surface 150, and the visible portions of thetangible content item 102 may be the projected content item that isvisually exposed to the video capture device 110 of the computing device100. Thus, the tangible content item 102 may be the content item that isdisplayed or otherwise physically present on the physical activitysurface 150 and can be captured by the video capture device 110 of thecomputing device 100.

FIG. 1B illustrates an example configuration in which the computingdevice 100 may provide a digital content item based on the contentdetected in the tangible content item 102. As shown in the example, thetangible content item 102 may be a book that includes information aboutairplanes. The book may include a first portion 104 that include textrelated to the content of the tangible content item 102, a secondportion 106 that includes a graphic related to the content of thetangible content item 102, and a third portion 108 that includes anothergraphic related to the content of the tangible content item 102. Thebook may be open to a page that is visible within a field of view of avideo capture device 110 of the computing device and the video capturedevice may capture an image and/or video stream of the open pageincluding the first portion 104, second portion 106, and/or thirdportion 108. A detector 304 of the computing device 100 may performimage recognition processes to detect one or more visually instructiveelements, such as the text, the graphic, or the additional graphicincluded in the portions of the open page of the tangible content item102.

In further implementations, the detector 304 may detect an identity ofthe open page, such as by detecting a page number and/or a page identityanchor visible within the field of view of the camera 110. The detector304 may then use the identity of the open page to identify concept ofthe tangible content item 102. The concept of the tangible content item102 may be a topic that the tangible content item 102 covers, such as atitle of the book, a title of the chapter the open page is includedwithin, a course subject of the book, a specific topic being presentedon the open page, etc. The concept of the tangible content item 102 maybe based on the context of the various information being presented, suchas a relationship between the content in the first portion 104, thesecond portion 106, and/or the third portion 108 of the tangible contentitem 102.

The activity application 214 of the computing device 100 may thenidentify one or more digital content items based on the concept of thetangible content item 102, the identity of the open page, and/or thetopic being presented on the open page. The digital content item may besupplemental content related to the content present on the open page andmay enhance the experience of the user as the interact with the tangiblecontent item 102. In the example shown in FIG. 1B, the digital contentitem 138 may include additional text related to “aircraft structure,”“principles of aerodynamics,” and/or “what happens if all engines failon a plane” as shown. These additional texts may be selectable linksembedded into the GUI of the display screen and when selected by theuser may direct the display screen to the link and display the contentsof the resulting selection. In further implementations, the GUI of thedisplay screen may automatically display the content in detail withoutrequiring the user to do any addition interactions with the digitalcontent item 138. In further implementations, the digital content item138 may also include videos and/or images of airplanes that may beautomatically displayed or may be presented as selectable links for theuser to interact with. By identifying the concept being presented in thetangible content item 102 and then presenting related digital contentitems 138 to the user, the user gains an enhanced experience of beingpresented both physical and digital information that is related withouthaving to navigate to the information themselves. This enhances theeducational opportunities by reducing distractions caused by having tofind the additional information on a student own. It further enhanceseducational opportunities by being able to specifically provide curatedcontent directly related to a specific page a student is working on thatmay be causing confusion or needing additional guidance.

FIG. 1C illustrates a configuration in which the computing device 100may be part of a visualization system to provide data enhancement forthe tangible content item 102 present on the physical activity surface.As depicted, the computing device 100 may be placed on the stand 140situated on the physical activity surface 150. The computing device 100may include the video capture device 110 with the camera adapter 130situated over the video capture device 110. In some embodiments, thecamera adapter 130 may redirect the field of view of the video capturedevice 110, thereby enabling the video capture device 110 to capture thevideo stream including the activity scene 132 of the physical activitysurface 150 located in front of the display of the computing device asdepicted in FIG. 1C. As a result of this configuration, the videocapture device 110 may capture the video stream of the activity scene132 that includes the tangible object 102 on the physical activitysurface 150 without the user manually capturing the video stream usingthe computing device 100. In some embodiments, as the user continues tointeract with various tangible content items 102, the activityapplication 214 implemented on the computing device 100 may perform themethod 400 in FIG. 4 for each tangible content item 104 with which theuser is currently interacting. As a result, the computing device 100 mayautomatically present the digital content items that are relevant to thetangible content item 102 in real-time. Thus, when the user interactswith a different tangible content item 102 in the physical activitysurface 150, the digital content items being displayed to the user onthe computing device 100 may be automatically updated accordingly.

In some implementations, the visualization system depicted in FIG. 1Cmay be an integrated system where the stand 140, computing device 100,and video capture device 110 are all integrated into housing of thecomputing device 100. In some implementations, the video capture device110 may be positioned so that the field of view encompass the physicalactivity surface without redirecting the field of view using the cameraadapter 130. In further implementations, multiple video capture devices110 may be included within the computing device 100 and each of thedifferent video capture devices 110 may be directed to capture adifferent field of view. For example, a first video capture device 110 amay be directed downward towards the physical activity surface 150 and asecond video capture device 110 b may be directed outward towards a userfacing the computing device 100 in order to capture a video stream thatincludes the user, such as the user's face and hand motions.

FIG. 1D illustrates a configuration in which the computing device 100may be part of a visualization system to provide data enhancement forthe tangible content item 102 present on the physical activity surface150. In the example, the tangible content item 102 may be one moretangible interface objects that are arranged on the physical activitysurface 150. The user may create a specific arrangement of the tangibleinterface objects and the detector 304 may be able to detect thearrangement over time, including when specific tangible interfaceobjects were present in a first image from a video stream compared to alater image from a video stream. By detecting the arrangement andmanipulation of the tangible interface objects, the detector 304 may beable to provide to the activity application 214 various events, such aswhen an arrangement is started being built, completed, as well as if thearrangement matches an expected arrangement of the activity application214 or if the arrangement is different than expected (e.g., thearrangement may be incorrect). The activity application 214 may then beable to retrieve and display one or more digital content items 146 basedon the arrangement and/or manipulation of the tangible content items onthe physical activity surface.

In one example, the tangible interface objects may be programming tilesthat include one or more visually instructive elements that can beidentified by the detector 304. Based on the visually instructiveelements, the detector 304 may be able to determine a tangibleidentifier of each of the programming tiles arranged on the physicalactivity surface 150. As a user arranges the programming tiles to depicta series of commands to be executed step by step based on the way theprogramming tiles are arranged, the activity application 214 may searchfor related digital content items 146 that are conceptually related tothe arrangement of the programming tiles. The activity application 214may then present the digital content items 146 to the user to enhancehow the user is arranging the programming tiles.

In the example in FIG. 1D, the activity application 214 may identifydigital content items 146 that are an instruction set for how to use theprogramming tiles together. The user may be able to view an animation ofwhat happens when two specific programming tiles are arranged in aspecific way and enhance their learning of the programming tiles andcommands. By providing this instruction automatically, without requiringthe user to find and display the digital content items 146, the user maybe able quickly resolve questions or confusion around how to use variousobjects or items, as well as receiving supplemental material on aconcept related to what they are interacting with in the physicalactivity scene. In some implementations, these digital content items maybe instruction sets, such as animations on how to arrange the differentprogramming tiles, tasks that can be presented for the user to complete,such as specific arrangements based on the detected tangible interfaceobjects, videos of other users interacting with the tangible interfaceobjects, a product description on a web page listing what items areincluded in a set of tangible interface objects the user is interactingwith, etc.

FIG. 2 is a block diagram illustrating an example computer system 200that is capable of providing data enhancement for tangible contentpresent on a physical activity surface. As depicted, the system 200 mayinclude computing devices 100 a . . . 100 n and servers 202 a . . . 202n communicatively coupled via a network 206. In FIG. 2 and the remainingfigures, a letter after a reference number, e.g., “100 a”, represents areference to the element having that particular reference number. Areference number in the text without a following letter, e.g., “100”,represents a general reference to instances of the element bearing thatreference number. It should be understood that the system 200 depictedin FIG. 2 is provided by way of example and that the system 200 and/orfurther systems contemplated by this present disclosure may includeadditional and/or fewer components, may combine components and/or divideone or more of the components into additional components, etc. Forexample, the system 100 may include any number of servers 202, computingdevices 100, or networks 206. As depicted in FIG. 2 , the computingdevice 100 may be coupled to the network 206 via the signal line 208 andthe server 202 may be coupled to the network 206 via the signal line204. The computing device 100 may be accessed by user 190.

The network 206 may include any number of networks and/or network types.For example, the network 206 may include, but is not limited to, one ormore local area networks (LANs), wide area networks (WANs) (e.g., theInternet), virtual private networks (VPNs), mobile (cellular) networks,wireless wide area network (WWANs), WiMAX® networks, Bluetooth®communication networks, peer-to-peer networks, other interconnected datapaths across which multiple devices may communicate, variouscombinations thereof, etc.

The computing device 100 may be a computing device that has dataprocessing and communication capabilities. In some embodiments, thecomputing device 100 may include a processor (e.g., virtual, physical,etc.), a memory, a power source, a network interface, and/or othersoftware and/or hardware components, such as front and/or rear facingcameras, display screen, graphics processor, wireless transceivers,keyboard, firmware, operating systems, drivers, various physicalconnection interfaces (e.g., USB, HDMI, etc.). In some embodiments, thecomputing device 100 may be coupled to and communicate with one anotherand with other entities of the system 200 via the network 206 using awireless and/or wired connection. As discussed elsewhere herein, thesystem 200 may include any number of computing devices 100 and thecomputing devices 100 may be the same or different types of devices(e.g., tablets, mobile phones, desktop computers, laptop computers,etc.).

As depicted in FIG. 2 , the computing device 100 may include the videocapture device 110, a detection engine 212, and one or more activityapplications 214. In some implementations, the computing device 100and/or the video capture device 110 may be equipped with the cameraadapter 130 as discussed elsewhere herein, while in other embodiments,the video capture device 110 may be directed to the appropriate field ofview without the camera adapter 130. In some embodiments, the cameraadapter 130 may include one or more optical elements (e.g., mirrorsand/or lenses) to adapt the standard field of view of the video capturedevice 110 of the computing device 100 to capture substantially and onlythe activity scene 132 of the physical activity surface 150 in front ofthe display of the computing device 100, although other implementationsare also possible and contemplated. To adapt the field of view of thevideo capture device 110, the mirrors and/or lenses of the cameraadapter 130 may be positioned at an angle to redirect and/or modify thelight being reflected from the physical activity surface 150 into thevideo capture device 110. In some embodiments, the camera adapter 130may include a slot adapted to receive an edge of the computing device100 and retain (e.g., secure, grip, etc.) the camera adapter 130 on theedge of the computing device 100. In some embodiments, the cameraadapter 130 may be positioned over the video capture device 110 todirect the field of view of the video capture device 110 toward thephysical activity surface 150.

In some embodiments, the detection engine 212 may detect and/orrecognize tangible objects located in the activity scene 132 of thephysical activity surface 150, and cooperate with the activityapplication(s) 214 to provide the user with a virtual experience thatincorporates in real-time the tangible objects and the user manipulationof the tangible objects in the physical environment. As an example, thedetection engine 212 may detect a tangible object located in theactivity scene 132 of the physical activity surface 150 and/or recognizevisually instructive elements, distinct visual elements, user markingsin the tangible content item of the tangible object, and cooperate withthe activity application(s) 214 to provide the user with digital contentitems that are relevant to the tangible content item on the physicalactivity surface 150. In another example, the detection engine 212 mayprocess the video stream captured by the video capture device 110 todetect and recognize a tangible object created by the user on theactivity scene 132. The activity application 214 may generate avisualization of the tangible object created by the user, and display tothe user a virtual scene in which an animated character may interactwith the visualization of the tangible object. The components andoperations of the detection engine 212 and the activity application 214are described in details with reference to at least FIGS. 3 and 4 .

The server 202 may include one or more computing devices that have dataprocessing, storing, and communication capabilities. In someembodiments, the server 202 may include one or more hardware servers,server arrays, storage devices and/or storage systems, etc. In someembodiments, the server 202 may be a centralized, distributed and/or acloud-based server. In some embodiments, the server 202 may include oneor more virtual servers that operate in a host server environment andaccess the physical hardware of the host server (e.g., processor,memory, storage, network interfaces, etc.) via an abstraction layer(e.g., a virtual machine manager).

The server 202 may include software applications operable by one or moreprocessors of the server 202 to provide various computingfunctionalities, services, and/or resources, and to send and receivedata to and from the computing devices 100. For example, the softwareapplications may provide the functionalities of internet searching,social networking, web-based email, blogging, micro-blogging, photomanagement, video/music/multimedia hosting/sharing/distribution,business services, news and media distribution, user account management,or any combination thereof. It should be understood that the server 202may also provide other network-accessible services.

In some embodiments, the server 202 may include a search engine capableof retrieving results that match one or more search criteria from a datastore. As an example, the search criteria may include an image and thesearch engine may compare the image to product images in its data store(not shown) to identify a product that matches the image. In anotherexample, the detection engine 212 and/or the storage 310 (e.g., see FIG.3 ) may request the search engine to provide information that matches aphysical drawing, an image, and/or a tangible object extracted from avideo stream.

It should be understood that the system 200 illustrated in FIG. 2 isprovided by way of example, and that a variety of different systemenvironments and configurations are contemplated and are within thescope of the present disclosure. For example, various functionalitiesmay be moved from a server to a client, or vice versa and someimplementations may include additional or fewer computing devices,services, and/or networks, and may implement various client orserver-side functionalities. In addition, various entities of the system200 may be integrated into a single computing device or system ordivided into additional computing devices or systems, etc.

FIG. 3 is a block diagram of an example computing device 100. Asdepicted, the computing device 100 may include a processor 312, a memory314, a communication unit 316, an input device 318, a display 320, andthe video capture device 110 communicatively coupled by a communicationsbus 308. It should be understood that the computing device 100 is notlimited to such and may include other components, including, forexample, those discussed with reference to the computing devices 100 inFIGS. 1A-1C, 2, 4, and 5 .

The processor 312 may execute software instructions by performingvarious input/output, logical, and/or mathematical operations. Theprocessor 312 may have various computing architectures to process datasignals including, for example, a complex instruction set computer(CISC) architecture, a reduced instruction set computer (RISC)architecture, and/or an architecture implementing a combination ofinstruction sets. The processor 312 may be physical and/or virtual, andmay include a single core or plurality of processing units and/or cores.

The memory 314 may be a non-transitory computer-readable medium that isconfigured to store and provide access to data to other components ofthe computing device 100. In some embodiments, the memory 314 may storeinstructions and/or data that are executable by the processor 312. Forexample, the memory 314 may store the detection engine 212, the activityapplications 214, and the camera driver 306. The memory 314 may alsostore other instructions and data, including, for example, an operatingsystem, hardware drivers, other software applications, data, etc. Thememory 314 may be coupled to the bus 308 for communication with theprocessor 312 and other components of the computing device 100.

The communication unit 316 may include one or more interface devices(I/F) for wired and/or wireless connectivity with the network 206 and/orother devices. In some embodiments, the communication unit 316 mayinclude transceivers for sending and receiving wireless signals. Forexample, the communication unit 316 may include radio transceivers forcommunication with the network 206 and for communication with nearbydevices using close-proximity connectivity (e.g., Bluetooth®, NFC,etc.). In some embodiments, the communication unit 316 may include portsfor wired connectivity with other devices. For example, thecommunication unit 316 may include a CAT-5 interface, Thunderbolt™interface, FireWire™ interface, USB interface, etc.

The display 320 may display electronic images and data output by thecomputing device 100 for presentation to the user 190. The display 320may include any display device, monitor or screen, including, forexample, an organic light-emitting diode (OLED) display, a liquidcrystal display (LCD), etc. In some embodiments, the display 320 may bea touch-screen display capable of receiving input from one or morefingers of the user 190. For example, the display 320 may be acapacitive touch-screen display capable of detecting and interpretingmultiple points of contact with the display surface. In someembodiments, the computing device 100 may include a graphic adapter (notshown) for rendering and outputting the images and data for presentationon display 320. The graphic adapter may be a separate processing deviceincluding a separate processor and memory (not shown) or may beintegrated with the processor 312 and memory 314.

The input device 318 may include any device for inputting informationinto the computing device 100. In some embodiments, the input device 318may include one or more peripheral devices. For example, the inputdevice 318 may include a keyboard (e.g., a QWERTY keyboard), a pointingdevice (e.g., a mouse or touchpad), a microphone, a camera, etc. In someimplementations, the input device 318 may include a touch-screen displaycapable of receiving input from the one or more fingers of the user 190.In some embodiments, the functionality of the input device 318 and thedisplay 320 may be integrated, and the user 190 may interact with thecomputing device 100 by contacting a surface of the display 320 usingone or more fingers. For example, the user 190 may interact with anemulated keyboard (e.g., soft keyboard or virtual keyboard) displayed onthe touch-screen display 320 by contacting the display 320 in thekeyboard regions using his or her fingers.

The detection engine 212 may include a calibrator 302 and a detector304. The components 212, 302, and 304 may be communicatively coupled toone another and/or to other components 214, 306, 310, 312, 314, 316,318, 320, and/or 110 of the computing device 100 by the bus 308 and/orthe processor 312. In some embodiments, the components 212, 302, and 304may be sets of instructions executable by the processor 312 to providetheir functionality. In some embodiments, the components 212, 302, and304 may be stored in the memory 314 of the computing device 100 and maybe accessible and executable by the processor 312 to provide theirfunctionality. In any of the foregoing implementations, these components212, 302, and 304 may be adapted for cooperation and communication withthe processor 312 and other components of the computing device 100.

The calibrator 302 includes software and/or logic for performing imagecalibration on the video stream captured by the video capture device110. In some embodiments, to perform the image calibration, thecalibrator 302 may calibrate the images in the video stream to adapt tothe capture position of the video capture device 110, which may bedependent on the configuration of the stand 140 on which the computingdevice 100 is situated. When the computing device 100 is placed into thestand 140, the stand 140 may position the video capture device 110 ofthe computing device 100 at a camera height relative to the physicalactivity surface and a tilt angle relative to a horizontal line.Capturing the video stream from this camera position may causedistortion effects on the video stream. Therefore, the calibrator 302may adjust one or more operation parameters of the video capture device110 to compensate for these distortion effects. Examples of theoperation parameters being adjusted include, but are not limited to,focus, exposure, white balance, aperture, f-stop, image compression,ISO, depth of field, noise reduction, focal length, etc. Performingimage calibration on the video stream is advantageous, because it canoptimize the images of the video stream to accurately detect the objectsdepicted therein, and thus the operations of the activity applications214 based on the objects detected in the video stream can besignificantly improved.

In some embodiments, the calibrator 302 may also calibrate the images tocompensate for the characteristics of the activity surface (e.g., size,angle, topography, etc.). For example, the calibrator 302 may performthe image calibration to account for the discontinuities and/or thenon-uniformities of the activity surface, thereby enabling accuratedetection of objects on the activity surface when the stand 140 and thecomputing device 100 are set up on various activity surfaces (e.g.,bumpy surface, beds, tables, whiteboards, etc.). In some embodiments,the calibrator 302 may calibrate the images to compensate for opticaleffect caused by the camera adapter 130 and/or the optical elements ofthe video capture device 110. In some embodiments, the calibrator 302may also calibrate the video capture device 110 to split its field ofview into multiple portions with the user being included in one portionof the field of view and the activity surface being included in anotherportion of the field of view of the video capture device 110.

In some embodiments, different types of computing device 100 may usedifferent types of video capture devices 110 that have different cameraspecifications. For example, the tablets made by Apple may use adifferent type of video capture device 110 from the tablets made byAmazon. In some embodiments, the calibrator 302 may use the camerainformation specific to the video capture device 110 of the computingdevice 100 to calibrate the video stream captured by the video capturedevice 110 (e.g., focal length, distance between the video capturedevice 110 to the bottom edge of the computing device 100, etc.). Thecalibrator 302 may also use the camera position at which the videocapture device 110 is located to perform the image calibration.

The detector 304 includes software and/or logic for processing the videostream captured by the video capture device 110 to detect the tangibleobjects present in the activity surface in the video stream. In someembodiments, to detect an object in the video stream, the detector 304may analyze the images of the video stream to determine line segments,and determine the object that has the contour matching the line segmentsusing the object data in the storage 310. In some embodiments, thedetector 304 may provide the tangible objects detected in the videostream to the activity applications 214. In some embodiments, thedetector 304 may store the tangible objects detected in the video streamin the storage 310 for retrieval by other components. In someembodiments, the detector 304 may determine whether the line segmentsand/or the object associated with the line segments can be identified inthe video stream, and instruct the calibrator 302 to calibrate theimages of the video stream accordingly. In some embodiments, thedetector 304 may perform character recognition (such as OCR as describedelsewhere herein) to identify visually instructive elements, such asword or reference marks that may be used to determine tangibleidentifiers of the tangible content item 102.

The activity application 214 includes software and/or logic executableon the computing device 100. In some embodiments, the activityapplication 214 may receive the characters, distinct visual elements,user markings in the tangible objects detected in the video stream ofthe activity surface from the detector 304. The activity application 214may determine one or more visually instructive elements (such askeywords or text, etc.) based on these factors, retrieve one or moredigital content items using the visually instructive elements, anddisplay the digital content items on the computing device 100.Non-limiting examples of the activity application 214 include videogames, learning applications, assistive applications, storyboardapplications, collaborative applications, productivity applications,etc. Other types of activity application are also possible andcontemplated.

The camera driver 306 includes software storable in the memory 314 andoperable by the processor 312 to control/operate the video capturedevice 110. For example, the camera driver 306 may be a software driverexecutable by the processor 312 for instructing the video capture device110 to capture and provide a video stream and/or a still image, etc. Insome embodiments, the camera driver 306 may be capable of controllingvarious features of the video capture device 110 (e.g., flash, aperture,exposure, focal length, etc.). In some embodiments, the camera driver306 may be communicatively coupled to the video capture device 110 andother components of the computing device 100 via the bus 308, and thesecomponents may interface with the camera driver 306 to capture videoand/or still images using the video capture device 110.

As discussed elsewhere herein, the video capture device 110 (alsoreferred to herein as a camera) is a video capture device adapted tocapture video streams and/or images of the physical activity surface. Insome embodiments, the video capture device 110 may be coupled to the bus308 for communication and interaction with the other components of thecomputing device 100. In some embodiments, the video capture device 110may include a lens for gathering and focusing light, a photo sensorincluding pixel regions for capturing the focused light, and a processorfor generating image data based on signals provided by the pixelregions. The photo sensor may be any type of photo sensor (e.g., acharge-coupled device (CCD), a complementary metal-oxide-semiconductor(CMOS) sensor, a hybrid CCD/CMOS device, etc.). In some embodiments, thevideo capture device 110 may include a microphone for capturing sound.Alternatively, the video capture device 110 may be coupled to amicrophone coupled to the bus 308 or included in another component ofthe computing device 100. In some embodiments, the video capture device110 may also include a flash, a zoom lens, and/or other features. Insome embodiments, the processor of the video capture device 110 maystore video and/or still image data in the memory 314 and/or provide thevideo and/or still image data to other components of the computingdevice 100, such as the detection engine 212 and/or the activityapplications 214.

In some embodiments, multiple video capture devices (such as 110 a and110 b) may be included in the computing device 100. These multiple videocapture devices 110 may include separate fields of view directed todifferent portions of the area around the computing device 100. In someembodiments, the fields of view of the multiple video capture devicesmay overlap and allow for stereo images or video streams to be createdby an activity application(s) 214, where the different two-dimensionalvideo streams may be combined to form a three-dimensional representationof an tangible content item 102 in a graphical user interface of thedisplay of the computing device 100.

The storage 310 is a non-transitory storage medium that stores andprovides access to various types of data. Non-limiting examples of thedata stored in the storage 310 include video stream and/or still imagescaptured by the video capture device 110, various calibration profilesassociated with each camera position of the video capture device 110,object data describing various tangible objects, image recognitionmodels and their model parameters, textual recognition models and theirmodel parameters, visually instructive elements corresponding to eachtangible content item, etc. In some embodiments, the storage 310 may beincluded in the memory 314 or another storage device coupled to the bus308. In some embodiments, the storage 310 may be or included in adistributed data store, such as a cloud-based computing and/or datastorage system. In some embodiments, the storage 310 may include adatabase management system (DBMS). The DBMS may be a structured querylanguage (SQL) DBMS. For example, the storage 310 may store data in anobject-based data store or multi-dimensional tables including rows andcolumns, and may manipulate (i.e., insert, query, update, and/or delete)data entries stored in the storage 310 using programmatic operations(e.g., SQL queries and statements or a similar database manipulationlibrary). Other implementations of the storage 310 with additionalcharacteristics, structures, acts, and functionalities are also possibleand contemplated. In some implementations, the storage 310 may include adatabase of previously scanned tangible content, such as books, that maybe scanned in and then the detector 304 may OCR the tangible content forlater identification and retrieval as described elsewhere herein.

An example method 400 for enhancing tangible content on a physicalactivity surface is illustrated in FIG. 4 . The method 400 may beperformed by a detector 304 and an activity application 214 implementedon the computing device 100 (see FIGS. 2 and 3 ). In block 402, thevideo capture device 110 of the computing device 100 may capture thevideo stream that includes the activity scene 132 of the physicalactivity surface 150. As discussed above, the physical activity surface150 may include a tangible content item 102 situated on the physicalactivity surface 150. The tangible content item 102 may be a tangibleobject that a user can place and manipulate in the physical activityscene 132. The tangible content item 102 may include one or moreportions that are visually exposed to the video capture device 110 ofthe computing device 100, and thus the video capture device 110 maycapture the tangible content item 102 in the video stream as depicted inFIG. 1A.

In block 404, the detector 304 may detect in the video stream thetangible content item 102 situated on the physical activity surface 150.In some embodiments, the detector 304 may apply an object detectionalgorithm to the image of the video stream to detect the tangiblecontent item 102 depicted in the image. For example, as depicted in FIG.1A, the detector 304 may detect the tangible content item 102 thatincludes textual content and/or graphical content in the image 112 ofthe video stream. In some implementations, this image 112 may bedisplayed as a live feed on a display of the computing device 100. Infurther implementations, the image 112 may be processed by the detector304 from the video stream and may not be displayed to the user.

In block 406, the detector 304 may recognize from the video stream oneor more visually instructive elements in the tangible content item 102.In some embodiments, the detector 304 may apply a textual recognitiontechnique (e.g., Optical Character Recognition—OCR) to the image of thevideo stream to detect the visually instructive elements in the tangiblecontent item 102. The visually instructive elements may include variousletters, characters, numbers, symbols, etc., that form the textualcontent of the tangible content item 102. In some embodiments, inaddition to the visually instructive elements, the detector 304 maydetect in the video stream one or more distinct visual elements in thetangible content item 102. Non-limiting examples of the distinct visualelement include images, drawings, diagrams, vision markers specificallydesigned to indicate certain information, etc. In some embodiments, thedetector 304 may apply an image recognition technique to the image ofthe video stream to determine the objects depicted in the distinctvisual element included in the tangible content item 102. For example,as depicted in FIGS. 1A and 1B, the detector 304 may apply the imagerecognition technique to the image 112 of the video stream. The detector304 may determine that the tangible content item 104 includes a diagram106 and a drawing 108, and determine that the diagram 106 depicts anairplane and the drawing 108 depicts an airplane turbine.

In some embodiments, the detector 304 may recognize from the videostream the metadata of the tangible content item 102. The metadata ofthe tangible content item 102 may provide general information about thetangible content item 102 and may be positioned on the top portionand/or the bottom portion of the tangible content item 104. In someembodiments, the detector 304 may apply the textual recognitiontechnique (e.g., OCR) on the top portion and/or the bottom portion ofthe tangible content item 102 to detect the metadata of the tangiblecontent item 102. Non-limiting examples of the metadata of the tangiblecontent item 102 include, but are not limited to, the document title ofthe tangible object 102 including the tangible content item 102 (e.g.,“Airplane Structure for Engineering Students”), the page numberassociated with the tangible content item 102 (e.g., pages 25 and 26),the section title associated with the tangible content item 102 (e.g.,“General Components of Airplane”), the section number associated withthe tangible content item 102 (e.g., Chapter 1), etc.

In some embodiments, the detector 304 may detect in the video stream oneor more user markings on the tangible content item 104. The usermarkings may be an emphasize effect, such as highlighting created by auser to emphasize a portion of the tangible content item 102 (e.g.,highlighting, underlining, etc.) or a user note added to a portion ofthe tangible content item 102 by the user (e.g., hand-writing text,sketch, diagram, etc.). In some embodiments, the detector 304 mayrecognize from the video stream one or more characters associated withthe user marking. For the user marking that is an emphasize effect, thedetector 304 may determine an emphasized portion of the tangible contentitem 102 indicated by the user marking, and apply the textualrecognition technique (e.g., OCR) on the emphasized portion of thetangible content item 104 to determine the visually instructive elementsassociated with the user marking. For the user marking that is a usernote added to the tangible content item 102 by the user, the detector304 may apply a machine learning model for handwriting recognition onthe user note to recognize the handwriting characters in the user note.The detector 304 may also apply the image recognition technique on theuser note to detect the distinct visual element in the user note (e.g.,sketch, diagram, etc.) and determine the objects depicted by thedistinct visual element in the user note. For example, the detector 304may detect a sketch of airflows created by the user around the body ofthe airplane in the tangible content item 102.

In some embodiments, the detector 304 may generate a detection resultincluding one or more of the visually instructive elements recognized inthe tangible content item 102, the distinct visual element included inthe tangible content item 102 and/or the objects detected in thedistinct visual element, the metadata of the tangible content item 102that is recognized in the tangible content item 102 (e.g., documenttitle, page number, section title, section number, etc.), the charactersassociated with the user markings in the tangible content item 102, thedistinct visual element associated with the user markings and/or theobjects detected in the distinct visual element, etc. In someembodiments, the detector 304 may transmit the detection result to theactivity application 214. The activity application 214 may be capable ofproviding additional content related to the tangible content item 104based on the detection result.

In block 408, the activity application 214 may determine one or moretangible identifiers based on the characters in the tangible contentitem 104. To determine the tangible identifiers, the activityapplication 214 may analyze the detection result generated by thedetector 304 to obtain the visually instructive elements recognized inthe tangible content item 102. The activity application 214 mayaggregate these recognized visually instructive elements into words orother categories, each word may include the characters located betweentwo sequential space characters in the tangible content item 102 In someembodiments, the activity application 214 may select one or moretangible identifiers from the words in the tangible content item 102,such as a keyword. For example, the activity application 214 maydetermine a heading line in the tangible content item 102, and selectone or more descriptive words in the heading line to be the tangibleidentifier. In another example, the activity application 214 maydetermine one or more emphasized words that are presented in differentformat from other words in the tangible content item 102 (e.g.,different font, size, color, etc.), and select the emphasized words tobe the tangible identifiers. In some embodiments, the activityapplication 214 may determine the descriptive words in the tangiblecontent item 102, compute a repeat frequency indicating a number oftimes each descriptive word is repeated in the tangible content item102, and select the descriptive words that have the repeat frequencysatisfying a repeat frequency threshold (e.g., more than seven times) tobe the tangible identifier.

In some embodiments, instead of using the words included in the tangiblecontent item 102, the activity application 214 may determine the contentcategory of the tangible content item 102 based on the descriptive wordsand/or the distinct visual element in the tangible content item 102, andselect the terminologies associated with the content category of thetangible content item 102 to be the visually instructive elements. As anexample, the activity application 214 may categorize the tangiblecontent item 102 into the content category “airplane,” and select theterminologies associated with the content category “airplane” to be thekeywords (e.g., “aircraft,” “aerodynamics,” “airplane engine,” etc.).

In some embodiments, the activity application 214 may determine thetangible identifier based on the distinct visual element included in thetangible content item 102 and/or the objects detected in the distinctvisual element. As an example, the activity application 214 maydetermine that the tangible content item 102 includes an image of anairplane engine. The activity application 214 may also determine theimage description of the image from the recognized characters of thetangible content item 102 (e.g., “Gas-turbine engine of jet aircraft”).In this example, the activity application 214 may determine the tangibleidentifier based on the airplane engine depicted in the image and theimage description of the image (e.g., “airplane engine,” “aircraftturbine,” “gas-turbine,” “jet aircraft,” etc.).

In some embodiments, the activity application 214 may determine thetangible identifier based on the metadata of the tangible content item102. The activity application 214 may analyze the detection resultgenerated by the detector 304 to obtain the metadata including thedocument title, the page number, the section title, and/or the sectionnumber associated with the tangible content item 102. If the sectiontitle associated with the tangible content item 102 is not included inthe detection result, the activity application 214 may retrieve thetable of content of the tangible object 102 that contains the tangiblecontent item 102 from the data storage 310 (see FIG. 3 ), and determinethe section title from the table of content of the tangible content item102 using the page number and/or the section number associated with thetangible content item 102. In some embodiments, the document titleand/or the section title associated with the tangible content item 102may provide a general summary of the content presented in the tangiblecontent item 102. Therefore, the activity application 214 may determinethe descriptive words in the document title and/or the section titleassociated with the tangible content item 102, and select thedescriptive words to be the tangible identifier.

In some embodiments, the activity application 214 may determine thetangible identifier based on the characters associated with the usermarking. The user marking may indicate the information that the userconsidered as important and should pay attention to in the tangiblecontent item 102. In some embodiments, the activity application 214 mayanalyze the detection result generated by the detector 304 to obtain thecharacters associated with the user marking. As discussed elsewhereherein, the characters associated with the user marking may be thecharacters in the portion of the tangible content item 102 that issubjected to the emphasize effect created by the user and/or thecharacters in the user note that is added to the tangible content item102 by the user. In some embodiments, the activity application 214 mayaggregate the characters associated with the user marking into words,each word may include the characters located between two sequentialspace characters. The activity application 214 may determine one or moredescriptive words among the words formed by the characters associatedwith the user marking, and select the descriptive words to be thetangible identifier. As an example, the user may highlight a word“turbulence modeling” and add a handwriting note “use k-omega model withwall function” in the tangible content item 102. In this example, theactivity application 214 may determine the descriptive words in the usermarkings, and determine the keywords to be “turbulence modeling,”“k-omega model,” and “wall function.”

In some embodiments, the activity application 214 may analyze thedetection result generated by the detector 304 to also obtain thedistinct visual element associated with the user markings and/or theobjects detected in the distinct visual element In some embodiments, theactivity application 214 may determine the keywords based on thedistinct visual element associated with the user markings and/or theobjects detected therein. As an example, the user may create a sketch ofan airplane with arrows illustrating airflows around the body of theairplane in the tangible content item 102. In this example, the activityapplication 214 may determine that the user note includes the airplaneand the airflows, and determine the keywords to be “airplane,”“airflows,” and “aerodynamic.”

In some embodiments, the tangible object may include a plurality oftangible content items 102 and the user may interact with differenttangible content items 102 of the tangible object. For example, thetangible content item 102 may be a book, and the user may read a firstportion of the first tangible content item 102 associated with pages 11and 12 of the book, and then turn to a second portion of the secondtangible content item 102 associated with pages 3 and 4 of the book. Insome embodiments, the activity application 214 may monitor the tangiblecontent items 102 with which the user interacted, and determine thetangible identifier and/or visually instructive elements based on thesetangible content items 102. To determine the tangible identifier, theactivity application 214 may determine a first tangible content item 102with which the user interacted at a first timestamp, and recognize oneor more visually instructive elements in the first tangible content item102. The activity application 214 may determine a second tangiblecontent item 102 with which the user interacted at a second timestamp,and recognize one or more visually instructive elements in the secondtangible content item 102. The first timestamp at which the userinteracted with the first tangible content item 102 may be prior to orsubsequent to the second timestamp at which the user interacted with thesecond tangible content item 102. In some embodiments, the time distancebetween the first timestamp and the second timestamp may satisfy a timedistance threshold (e.g., less than 30 s).

In some embodiments, the activity application 214 may determine thetangible identifier based on the characters in the first tangiblecontent item 102 and the characters in the second tangible content item102. In addition to the characters, the activity application 214 mayalso determine the tangible identifier based on the distinct visualelements, the user markings, etc., in the first tangible content item102 and the second tangible content item 102 as discussed above. In someembodiments, the activity application 214 may select the descriptivewords included in both the first tangible content item 102 and thesecond tangible content item 102 to be the tangible identifiers.Alternatively, the activity application 214 may select the tangibleidentifiers that are relevant to the content of the first tangiblecontent item 102 and the content of the second tangible content item102. For example, the first tangible content item 102 may presentinformation about airflows around airplane body, and the second tangiblecontent item 102 may present information about shape and size ofcommercial airplanes. In this example, the activity application 214 maydetermine the tangible identifier to be “airflows around commercialairplanes.” Determining the tangible identifier based on multipletangible content items 102 with which the user interacted isadvantageous, because this implementation can increase the likelihood ofthe tangible identifier being relevant to the content that the user isinterested in.

In block 410, the activity application 214 may retrieve one or moredigital content items using one or more tangible identifiers. In someembodiments, the activity application 214 may query the data storage 310and/or other content databases using the tangible identifier to retrievethe digital content items. The digital content items returned as thequery result may include the tangible identifiers, and thus may berelevant or related to the tangible content item 102 that is present onthe physical activity surface 150. Non-limiting examples of the digitalcontent item includes images, videos, audios, webpages, electronicdocuments, instructions, etc. Other types of digital content item arealso possible and contemplated.

In some embodiments, the activity application 214 may retrieve thedigital content items using one tangible identifier in the query.Alternatively, the activity application 214 may retrieve the digitalcontent items using multiple tangible identifiers. In some embodiments,the activity application 214 may determine a first tangible identifierand a second tangible identifier based on the recognized visuallyinstructive elements, the distinct visual elements, the user markings,etc., in the tangible content item 102 as discussed above. In someembodiments, the activity application 214 may determine a contextualrelationship between the first tangible identifier and the secondtangible identifier in describing subject matters similar to thetangible content item 102. For example, the contextual relationshipbetween the first tangible identifier and the second tangible identifiermay indicate the likelihood of the first tangible identifier and thesecond tangible identifier being in the same content item that belongsto the content category of the tangible content item 102, the averagedistance between the first tangible identifier and the second tangibleidentifier in the content item, etc. In some embodiments, the contextualrelationship between the first tangible identifier and the secondtangible identifier may be indicated by a contextual relationshipmetric.

In some embodiments, the activity application 214 may retrieve thedigital content items based on the contextual relationship between thefirst tangible identifier and the second tangible identifier. In someembodiments, the activity application 214 may determine whether thecontextual relationship metric between the first tangible identifier andthe second tangible identifier satisfies a contextual relationshipmetric threshold (e.g., more than 0.75). If the contextual relationshipmetric between the first tangible identifier and the second tangibleidentifier satisfies the contextual relationship metric threshold, theactivity application 214 may retrieve the digital content items usingboth the first tangible identifier and the second tangible identifier.For example, the activity application 214 may query the data storage 310and/or other content databases using the first tangible identifier andthe second tangible identifier in the same query to retrieve the digitalcontent items. The digital content items returned as the query resultmay include both the first tangible identifier and the second tangibleidentifier. In some embodiments, the relative distance between the firsttangible identifier and the second tangible identifier in the digitalcontent items may be proportional to the contextual relationship metricbetween the first tangible identifier and the second tangibleidentifier.

In some embodiments, the activity application 214 may retrieve thedigital content item from a knowledge graph stored in the storage 310.The knowledge graph may include one or more nodes related to differenttopics and each node may be linked to other nodes related to differenttopics within the knowledge graph. Each node may include digital contentitems related to the topic of that node in the knowledge graph. In someembodiments, the knowledge graph may link sub-nodes with sub-topics tothe main node with a main-topic in the knowledge graph and as theactivity application 214 provides the one or more tangible identifiersto the knowledge graph, the nodes related to those tangible identifiersmay be identified and digital content items within that node may beretrieved. In some embodiments, the nodes may also include categorylevels for different users, where the different category levels relateto depth of the knowledge on the topic. For example, if the node of theknowledge graph is related to aerodynamics, a user with a lower categorylevel (such as in a second year of school) will be different than a userwith a higher category level (such as in a college level year ofschool). Based on the category level of a user, different digitalcontent items related to the topic of the node may be provided that arerelevant to that user based on the category level.

In block 412, once the digital content items are retrieved, the activityapplication 214 may provide the digital content items on the computingdevice 100. The digital content items may include videos, images,webpages, electronic documents, etc., displayed on the computing device100, and may provide content data relevant to the tangible content item104 of the tangible object 102 that is present on the physical activitysurface 150. An example of the digital content items being provided onthe computing device 100 is illustrated in FIG. 1B. In this example, thetangible content item 104 on the physical activity surface 150 mayinclude the characters, the distinct visual elements, and/or the usermarkings related to airplane, and the digital content items displayed onthe computing device 100 may present content data relevant to airplane(e.g., airplane parts, aircraft structure, principle of aerodynamics,airplane engine, etc.) as depicted in FIG. 1B.

FIG. 5 is an example configuration 500 for enhancing tangible content ona physical activity surface. As shown in the example, the computingdevice 100 may include a first camera 110 a that includes a first fieldof view 502 directed downwards towards a surface of in front of thecomputing device 100. The computing device 100 may also include a secondcamera 110 b that includes a second field of view 504. The second fieldof view 504 may be directed outward from the computing device 100 andinclude the area in front of the computing device 100. In someimplementations, the second field of view 504 may be able to capture animage of at least a portion user interacting with the physical activitysurface in front of the computing device 100. For example, the secondfield of view 504 may be able to capture an upper torso of a first userincluding their head and arms. This may allow the activity application214 to view facial expressions and/or hand gestures performed by theuser in the second field of view 504. In some implementations, thesecond field of view 504 may be displayed on the computing device 100 oranother computing device (not shown). In this example, the displayscreen may display multiple items on the display screen, one of moreitems may be based on the physical content items 102 included in thefirst field of view 502 and another item may be a video stream of theuser from the second field of view 504.

In some implementations, the activity application 214 may be able to usethe second field of view 504 to identify a specific user present in thevideo stream. For example, the activity application 214 may identify afirst user as being in a first grader and a second user as being aparent of the first user and may present appropriate content based onthe identity of the user. If the first user is detected in the secondfield of view 504, then the activity application may retrieve thecurrent homework tasks for the first grader and present them on thedisplay screen for the first grader to begin working on. If the seconduser is detected in the second field of view 504, then the activityapplication may display a status of the various tasks that have beenassigned to the first grader and easily facilitate helping the parentknow how the first grader is doing on various tasks, without having tonavigate to specific content manually.

In some implementations, the activity application 214 may use one ormore of the fields of view of the camera 110 to execute variousapplications. For example, when no user is detected in a field of viewof the camera 110, then the activity application 214 may cause thecomputing device 100 to go into a passive or sleep mode. In someimplementations, in the passive mode, the activity application 214and/or the detector 304 may routinely check the fields of view of thecameras 110 and if something is detected in one or more of the fields ofview, then the activity application 214 may execute an appropriateprogram. For example, the detector 304 may detect a user positioningthemselves (such as by sitting down in a chair in front of the computingdevice 100) and may cause the activity application 214 to determine anidentity of the user and/or one or more relevant applications to beingdisplaying on the display screen.

For example, when a first user is detected, then activity application214 may retrieve a profile of the first user and review commonlyexecuted applications. The activity application 214 may then launch oneor more of these applications before receiving a selection by the user.These applications may be launched in the background without the user'sexpress knowledge and if the user requests to run one of thepre-launched applications, then the activity application 214 may displaythe pre-launched application and/or close down the other applications.This allows the activity application 214 to quickly execute applicationsand the users doesn't have to wait for common applications to load upbased on the profile of the identified user.

In some implementations, the detector 304 may detect an object, such astangible content item 102 being positioned in one of the fields of viewof the camera 100 and may cause the activity application 214 to retrieveone or more digital content items based on a tangible identifier of thetangible content item 102. The activity application 214 may furtherdisplay a graphic and/or animation on the display screen to guide theuser on how to begin using the tangible content item 102, such asdisplaying a current page number for the user to open the tangiblecontent item 102, etc. In some implementations, the detector 304 maysense the presence of a user and/or tangible content item 102 anddisplay a query, prompt, or detection awareness graphic to signal to theuser that they and/or the object have been detected and can begininteracting with the computing device 100.

FIG. 6 is a flowchart 600 of an example method for enhancing tangiblecontent on a physical activity surface. At 602, the video capture devicemay capture a video stream that includes an activity scene 132 of aphysical activity surface 150. In some implementations, as describedelsewhere herein, the computing device 100 may be positioned on thephysical activity surface 100, such as in a stand or an integrateddevice. In further implementations, the computing device 100 may beportable and positioned by a user 100 to capture a video stream of thephysical activity surface 150.

At 604, the detector 304 may detect in the video stream one or moretangible content items 102 included in the video stream. In someimplementations, the tangible content item 102 may include one or morepages that have been opened to be visible within the field of view ofthe video capture device. In further implementations, the tangiblecontent item 102 may be a single page, such as a worksheet, that isvisible within the field of view of the video capture device.

At 606, the activity application 214 may determine an identity of thepage of the tangible content item 102 from the video stream. Theidentity of the page may be determined based on one or more visuallyinstructive elements included on the page, such as a page number, atitle, a graphic, etc. In further implementations, the identity of thepage may be determined based on a page identity anchor on the page thatmay be a unique graphic or textual icon visible to the detector 304. Theactivity application 214 may determine the identity of the page based onthe page identity anchor or other visually instructive elements includedwithin the page. For example, each page of a tangible content item 102may have a unique page identity anchor and each page identity anchor maymatch a database of page identity anchors that are associated withdifferent pages. In some implementations, pages of the tangible contentitem 102 may be captured as images and image recognition may beperformed on the pages to determine content of the page and one or moreconcepts the pages are related to. The pages may then be catalogued in adatabase for future retrieval when their unique page identity anchor hasbeen identified. In some implementations, the detector 304 may beconfigured to only search for page identity anchors in the imagescaptured by the video stream to increase the speed at which the activityapplication 304 can identify the page. By only looking for the pageidentity anchor, the detector 304 can quickly ignore the other textand/or graphics present on a tangible content item 102 until the pageidentity anchor has been identified and the profile of the page has beenretrieved.

At 608, the activity application 214 may determine a title of thetangible content item using the identity of the page. For example, theactivity application 214 may know that the page is page 26 of a specificbook and the tangible content item 102 may be the specific book based onthis identity of the page 26. The title of the book may then signal thesubject and/or context of the tangible content item 102.

At 610, the activity application 214 may determine digital content itemsbased on the title of the tangible content item and/or the identity ofthe page of the tangible content item 102. For example, where thetangible content item 102 is a textbook on math title “3 ^(rd) GradeMath” and the specific page identity is page 26 related to linesegments. The activity application 214 may use the contextualinformation related to the title “3^(rd) Grade Math” and the “linesegments” to identify a digital content item representing a video of howto solve a problem with line segments by a different third gradeteacher. In some implementations, the digital content item can be one ormore of a video, a question, a quiz, a worksheet, etc. At 612, thedigital content item is then presented in the graphical user interfaceof the display screen. This supplemental digital content item can bepresented on the display of the computing device 100 for the user towatch and assist in the learning they are accomplishing.

FIGS. 7A-7D are graphical representations of an example user interfacefor enhancing tangible content. As shown in FIG. 7A, the graphicalrepresentation 700 depicts a user interface 702 that includes apersonalization portion 708. The personalization portion may be based ona detected identity of a user in a field of view of the camera 110. Thepersonalization portion may include various settings that the user maypersonalize to create their own experience.

The user interface 702 may include progression icons 704 displaying aprogress of the specific user through various topics. For example,progression icon 704 a depicts a progress in “Practical GeometryMathematics” and includes a progress bar showing the majority of thetopics related to that progression icon 704 a have been completed. Asdifferent tasks or topics are assigned to a user, such as by a teacheras homework or supplemental learning, the progression icons 704 mayupdate to include an additional topic based on the new tasks. Theprogression icons may be selectable by a user and may include nestedicons representing different subtopics within the progression icons. Theprogressions icons may be displayed in an easily accessible area for theuser to begin selecting topics quickly. In some implementations, theactivity application 214 may cause one of the topics of the progressionicons 704 to be displayed before a user begins interacting with thedisplay screen.

The user interface 702 may include topic icons 706 representingdifferent topics and/or tasks for the user to interact with. Forexample, the user may be able to select between “practical geometry” intopic icon 706 a or “stars and the solar system” in topic icon 706 c.The topic icons 706 may extend beyond an edge of the screen and the usermay be able to scroll through the various topic icons 706 to selectvarious icons.

As shown in FIG. 7B, the graphical representation 710 may update theuser interface 702 when the activity application detects a tangiblecontent item 102 (not shown) has been positioned within a field of viewof the camera 100. For example, a user may place a text book related to“Atoms and Molecules” on the physical activity surface and the activityapplication 214 may cause a detected textbook icon 712 to be displayedon the user interface 702. In some implementations, the detectedtextbook icon 712 may scroll into the user interface 702 so that itappears it is being pushed into the field of view as the user moves thephysical textbook related to “Atoms and Molecules” into the field ofview of the camera 110. By imitating the physical placement of thetangible content item 102, the detected textbook icon 712 may create amore immersive experience for the user. In further implementations, asdifferent textbooks are moved on positioned on the physical activitysurface 150, the user interface 702 may cause the virtualrepresentations of the topic icons 706 or detected textbook icons 712 tomirror those movements and move in and out of the user interface 702similar to how the physical items are moving in and out of the field ofview of the camera 102 in order to create a virtual tabletop that mimicswhat is present in the physical activity surface 150.

In some implementations, the activity application 214 may generate thedetected textbook icon 712 as a visualization based on a digital contentitem that may be appearing in the graphical user interface 702. Thedetected textbook icon 702 may be displayed as an animation of thedigital content item appearing in the user interface. In someimplementations, this animation may mimic the appearance of acorresponding tangible content item 102 that is detected by the detector304 as it appears in the video stream of the physical activity surface.In some implementations, the animation may start by displaying only afirst portion of the visualization of the detected textbook icon 712,such as shown in FIG. 7B, where the first portion of the detectedtextbook icon 712 c is displayed and a second portion is not shown inthe user interface. The animation may then appear to scroll the rest ofthe visualization of the detected textbook icon 712 to appear on thescreen, such as by presenting the second portion of the detectedtextbook icon 712 c as shown in FIG. 7C.

As shown in FIG. 7C, the graphical representation 714 may display theuser interface 702 where the detected textbook icon 712 is fullydisplayed in the user interface 712. In some implementations, the topicicons 706 a-706 c may be shifted to allow for the detected textbook icon712 to be displayed. In some implementations, the user interface 702 mayalso cause one or more subtopics, such as the portion of the detectedtextbook icons 712 a-712 b to be displayed along with an image of thefull topic of the detected textbook icon 712 c.

As shown in FIG. 7D, the graphical representation 716 may display theuser interface 702 where a user is navigating into one or more of thedetected textbook icons 712 as shown to display task icons 720 relatedto the selected topic 718. Using this nested topic option, a user canquickly navigate through various tasks and assignments based on topicand each of the topics can be trigged for display based on the detectionof a tangible content item 102 in the field of view of the camera 110.This automatic appearance of digital content based on the interactionwith a tangible content item 102 allows users to quickly access variousrelevant information. For example, a student can quickly view varioustasks and assignments for the day, just be viewing the progression icons704 or the user can place one of their textbooks or activity books intothe field of view of the camera 110 and it will automatically displaythe current assignments related to that book for the student to beginworking on.

FIG. 8 is an example configuration 800 for finger detection to enhancetangible content. As shown in the example, a computing device 100 may bepositioned on an activity surface with one or more cameras 110. Thedetector 304 may detect an appearance of a hand 804 and/or finger of auser in an image of the physical activity surface. The detector 304 mayidentify the hand 804 and/or finger based on expected contours or shapesdescribed elsewhere herein related to finger and hand shapes. In someimplementations, the detector 304 may be configured to identify a pointof a finger of the hand 804 within the image and may determine an areaof the tangible content item 102 that the point of the finger of thehand 804 is pointing to. As shown in FIG. 8 , the area 806 of thetangible content item 102 may be a portion of the tangible content item102 proximate to where the detector 304 has detected the point of thefinger of the hand 804 to be pointing. In some implementations, theactivity application 214 may map the position of the tangible contentitem 102 into a coordinate plane and divide the mapped tangible contentitem 102 into various areas within each section of the coordinate plane.The activity application 214 may map the position of the point of thefinger of the hand 804 into the same coordinate plane relative to theposition of the tangible content item 102 and identify which areas ofthe coordinate plane include both the point of the finger and thecorresponding area of the tangible content item 102.

In some implementations, the area 806 that the finger is pointing to ofthe tangible content item 102 may be analyzed by the detector 304 toidentify one or more tangible identifiers within the area. For example,a tangible content item 102 may be a textbook and the user may bepointing to an area 806 with their finger to a specific word or image onthe page. Using the finger detection, the detector 304 may focus on onlythe content within the area 806 to identify a tangible identifier, suchas only the specific word or image being pointed to by the point of thefinger. This allows the interaction between the tangible content item102 and the digital content items to be enhanced where a user can simplypoint to specific areas that they need assistance with or supplementalcontent provided for. For example, a user can be reading a book and notknow the definition for a specific word. The user can point at the wordand the activity application 214 may cause the definition of the word toappear and/or be audibly output using the computing device 100. Inanother example, when a user is stuck on a math problem, they can pointto a specific step in the math problem and the user can quickly providesuggestions for that specific step rather than the context of the entiremath problem.

It should be understood that the above-described example activities areprovided by way of illustration and not limitation and that numerousadditional use cases are contemplated and encompassed by the presentdisclosure. In the above description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the present disclosure. However, it should beunderstood that the technology described herein may be practiced withoutthese specific details. Further, various systems, devices, andstructures are shown in block diagram form in order to avoid obscuringthe description. For instance, various implementations are described ashaving particular hardware, software, and user interfaces. However, thepresent disclosure applies to any type of computing device that canreceive data and commands, and to any peripheral devices providingservices.

In some instances, various implementations may be presented herein interms of algorithms and symbolic representations of operations on databits within a computer memory. An algorithm is here, and generally,conceived to be a self-consistent set of operations leading to a desiredresult. The operations are those requiring physical manipulations ofphysical quantities. Usually, though not necessarily, these quantitiestake the form of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout this disclosure, discussions utilizingterms including “processing,” “computing,” “calculating,” “determining,”“displaying,” or the like, refer to the action and processes of acomputer system, or similar electronic computing device, thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system's registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

Various implementations described herein may relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a computer readable storage medium, including, but is notlimited to, any type of disk including floppy disks, optical disks,CD-ROMs, and magnetic disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flashmemories including USB keys with non-volatile memory or any type ofmedia suitable for storing electronic instructions, each coupled to acomputer system bus.

The technology described herein can take the form of a hardwareimplementation, a software implementation, or implementations containingboth hardware and software elements. For instance, the technology may beimplemented in software, which includes but is not limited to firmware,resident software, microcode, etc. Furthermore, the technology can takethe form of a computer program product accessible from a computer-usableor computer-readable medium providing program code for use by or inconnection with a computer or any instruction execution system. For thepurposes of this description, a computer-usable or computer readablemedium can be any non-transitory storage apparatus that can contain,store, communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.

A data processing system suitable for storing and/or executing programcode may include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories that provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution. Input/output or I/Odevices (including but not limited to keyboards, displays, pointingdevices, etc.) can be coupled to the system either directly or throughintervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems,storage devices, remote printers, etc., through intervening privateand/or public networks. Wireless (e.g., Wi-Fi™) transceivers, Ethernetadapters, and modems, are just a few examples of network adapters. Theprivate and public networks may have any number of configurations and/ortopologies. Data may be transmitted between these devices via thenetworks using a variety of different communication protocols including,for example, various Internet layer, transport layer, or applicationlayer protocols. For example, data may be transmitted via the networksusing transmission control protocol/Internet protocol (TCP/IP), userdatagram protocol (UDP), transmission control protocol (TCP), hypertexttransfer protocol (HTTP), secure hypertext transfer protocol (HTTPS),dynamic adaptive streaming over HTTP (DASH), real-time streamingprotocol (RTSP), real-time transport protocol (RTP) and the real-timetransport control protocol (RTCP), voice over Internet protocol (VOIP),file transfer protocol (FTP), WebSocket (WS), wireless access protocol(WAP), various messaging protocols (SMS, MMS, XMS, IMAP, SMTP, POP,WebDAV, etc.), or other known protocols.

Finally, the structure, algorithms, and/or interfaces presented hereinare not inherently related to any particular computer or otherapparatus. Various general-purpose systems may be used with programs inaccordance with the teachings herein, or it may prove convenient toconstruct more specialized apparatus to perform the required methodblocks. The required structure for a variety of these systems willappear from the description above. In addition, the specification is notdescribed with reference to any particular programming language. It willbe appreciated that a variety of programming languages may be used toimplement the teachings of the specification as described herein.

The foregoing description has been presented for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the specification to the precise form disclosed. Manymodifications and variations are possible in light of the aboveteaching. It is intended that the scope of the disclosure be limited notby this detailed description, but rather by the claims of thisapplication. As will be understood by those familiar with the art, thespecification may be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. Likewise, theparticular naming and division of the modules, routines, features,attributes, methodologies and other aspects are not mandatory orsignificant, and the mechanisms that implement the specification or itsfeatures may have different names, divisions and/or formats.

Furthermore, the modules, routines, features, attributes, methodologiesand other aspects of the disclosure can be implemented as software,hardware, firmware, or any combination of the foregoing. Also, whereveran element, an example of which is a module, of the specification isimplemented as software, the element can be implemented as a standaloneprogram, as part of a larger program, as a plurality of separateprograms, as a statically or dynamically linked library, as a kernelloadable module, as a device driver, and/or in every and any other wayknown now or in the future. Additionally, the disclosure is in no waylimited to implementation in any specific programming language, or forany specific operating system or environment. Accordingly, thedisclosure is intended to be illustrative, but not limiting, of thescope of the subject matter set forth in the following claims.

1. A method comprising: capturing, using a video capture device of acomputing device, a video stream that includes an activity scene of aphysical activity surface located in front of a display of the computingdevice; detecting in the video stream, using a detector executable onthe computing device, a first tangible interface object on the physicalactivity surface and a second tangible interface object on the physicalactivity surface; determining a first subject matter of the firsttangible interface object; determining that a user interacted with thesecond tangible interface object; determining a second subject matter ofthe second tangible interface object responsive to the user interactingwith the second tangible interface object; recognizing, from the videostream, one or more visually instructive elements of the first tangibleinterface object; determining a concept of the one or more visuallyinstructive elements of the first tangible interface object based on therecognized one or more visually instructive elements of the firsttangible interface object, the subject matter of the first tangibleinterface object, and the subject matter of the second tangibleinterface object; automatically retrieving a digital content item usingthe determined concept of one or more visually instructive elements ofthe first tangible interface object; and providing the digital contentitem for display in a user interface of the computing device tosupplement the one or more visually instructive elements of the firsttangible interface object on the physical activity surface.
 2. Themethod of claim 1, further comprising: detecting, in the video stream, adistinct visual element on the first tangible interface object; andwherein determining the concept of the one or more visually instructiveelements of the first tangible interface object includes determining theconcept of the one or more visually instructive elements of the firsttangible interface object based on the distinct visual element on thefirst tangible interface object.
 3. The method of claim 2, wherein: thedistinct visual element includes one or more of an image, a drawing, adiagram, and a vision marker visible on the first tangible interfaceobject.
 4. The method of claim 1, further comprising: recognizing, fromthe video stream, one or more of a document title, a page number, asection title, and a section number associated with the first tangibleinterface object; and wherein determining the concept of the one or morevisually instructive elements of the first tangible interface objectincludes determining the concept of the one or more visually instructiveelements of the first tangible interface object based on one or more ofthe document title, the page number, the section title, and the sectionnumber associated with the first tangible interface object.
 5. Themethod of claim 1, further comprising: detecting, in the video stream, auser marking visible on the first tangible interface object;recognizing, from the video stream, one or more characters associatedwith the user marking; and wherein determining the concept of the one ormore visually instructive elements of the first tangible interfaceobject includes determining the concept of the one or more visuallyinstructive elements of the first tangible interface object based on theone or more characters associated with the user marking.
 6. The methodof claim 5, wherein: the user marking includes one or more of ahighlighting effect and a user note created by a user on the firsttangible interface object.
 7. The method of claim 1, wherein determiningthe second subject matter of the second tangible interface objectfurther comprises: recognizing one or more visually instructive elementsin the second tangible interface object and determining the secondsubject matter of the second tangible interface object based on the oneor more visually instructive in the second tangible interface object. 8.The method of claim 1, wherein: determining the concept of the one ormore visually instructive elements of the first tangible interfaceobject includes determining a first concept and a second concept basedon the one or more visually instructive elements in the first tangibleinterface object; and retrieving the digital content item using thedetermined concept includes: determining a contextual relationshipbetween the first concept and the second concept; and retrieving thedigital content item based on the contextual relationship between thefirst concept and the second concept.
 9. The method of claim 1, wherein:the computing device is placed on a stand situated on the physicalactivity surface; and a field of view of the video capture device of thecomputing device is redirected by a camera adapter situated over thevideo capture device of the computing device.
 10. The method of claim 1,wherein the first tangible interface object on the physical activitysurface includes an area with a first tangible content section and asecond tangible content section, the method further comprising:detecting, using a detector executable on the computing device, anarrangement of the first tangible content section relative to the secondtangible content section within the area on the physical activitysurface; determining a supplemental digital content item based on thearrangement of the first tangible content section relative to the secondtangible content section; and displaying in the user interface on thecomputing device, the supplemental digital content along with thedigital content item.
 11. The method of claim 10, wherein the firsttangible content section is a first programming tile and the secondtangible content section is a second programming tile, the firstprogramming tile being coupled with the second programming tile.
 12. Themethod of claim 11, wherein the supplemental digital content is aninstruction set for how to use the first programming tile and the secondprogramming tile.
 13. The method of claim 12, wherein displaying thesupplemental digital content further comprises: automatically displayinga first digital representation of the first programming tile in the userinterface; automatically displaying a second digital representation ofthe second programming tile in the user interface; and automaticallydisplaying an animation of the first digital representation interactingwith the second digital representation to provide instructions on how toarrange the first digital programming tile and the second digitalprogramming tile on the activity surface.
 14. The method of claim 1,wherein providing the digital content item for display in the userinterface on the computing device further comprises: generating avisualization of the digital content item for display in the userinterface; and displaying an animation of the digital content itemappearing in the user interface.
 15. The method of claim 14, wherein theanimation of the digital content item depicts a first portion of thevisualization of the digital content item extending out from a bottomside of the display screen and a second portion of the visualization ofthe digital content item appearing as the animation causes thevisualization of the digital content item to move into the userinterface.
 16. The method of claim 15, wherein the animation of thedigital content item corresponds to an appearance of the first tangibleinterface object being detected on the physical activity surface. 17.The method of claim 14, wherein the visualization of the digital contentitem is selectable by a user and responsive to the digital content itembeing selected, the digital content item may be displayed.
 18. A methodcomprising: capturing, using a video capture device of a computingdevice, a video stream that includes an activity scene of a physicalactivity surface; detecting in the video stream, using a detectorexecutable on the computing device, a book on the physical activitysurface and a tangible interface object on the physical activity scene,the book including a page visible within a field of view of the videocapture device; determining, using a processor of the computing device,an identity of the page of the book from the video stream; determining afirst subject matter of the book using the identity of the page of thebook; determining that a user interacted with the tangible interfaceobject; determining a second subject matter of the tangible interfaceobject responsive to determining that the user interacted with thetangible interface object; determining, using the processor of thecomputing device, a concept of the page of the book using the identityof the page; determining, using the processor of the computing device, adigital content item based on the concept of the page of the book, thefirst subject matter of the book, and the second subject matter of thetangible interface object; and presenting, in a display of the computingdevice, the digital content item in a graphical user interface.
 19. Themethod of claim 18, wherein determining the identity of the page of thebook further comprises: detecting a page identity anchor on the page;and matching the page identity anchor to a database of page identities.20. The method of claim 19, wherein the book includes a plurality ofpages and each of the plurality of pages has a unique page identityanchor.
 21. (canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled) 25.(canceled)
 26. (canceled)
 27. (canceled)
 28. (canceled)